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(57) Abstract: Method for establishing a virtual three-dimensional representation of a bone or bone fragment from X-ray images 
comprising the steps of A) taking at least one medical image (1 1) of a patient' s bone (21) or bone fragment; B) storing the at least one 
^ medical image (1 1) as a set of data in a data storage means (9) of a computer (10); C) display the at least one medical image (1 1) at a 
display means (32) connected to the computer (10); D) creating a three-dimensional virtual representation of the bone (21) or bone 
£^ fragment by means of using a control means (26) connected to the computer (10); and E) adjusting the size of the three-dimensional 
)^ virtual representation to the size of the bone (21) or bone fragment by means of using the control means (26). 
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Method for establishing a three-dimensional representation of bone x-ray 
images 

The invention relates to a method for establishing a virtual three-dimensional 
representation of a bone or bone fragment from X-ray images according to the concept 
of claim 1 . 

In order to perform minimal invasive surgery quite often computer assisted surgery 
systems (CAS systems) or image guided surgery systems that are provided with a 
computer and a position measurement device in order to measure the position of 
surgical instruments, devices and a body portion of the patient are used. 

Such computer assisted surgery systems (CAS-systems) are disclosed e.g. in EP 0 359 
773 SCHLONDORFF, US 5,383,454 BUCHHOLZ and in US 5,682,886 DELP. Often 
these CAS - systems comprise a memory means apt to store medical images such as 
e.g. X-rays, Computertomographs or MR images (Magnetic Resonance images). 
Thereby the medical images may be gathered pre-operatively or intraoperatively 

Currently, there are three classes of computer assisted orthopaedic surgery systems 

a) CT based systems, which use a preoperative CT (Computertomogram) of a bone or 
bone fragment to establish a three-dimensional anatomical model that is referenced 
with the intraoperative respective bone or bone fragment through landmark based or 
surface based registration or matching; 

b) CT based and fluoroscopy systems, which use the same method as CT based 
systems to establish a three-dimensional anatomical model, whereby the preoperative 
CT of a bone or bone fragment is registered or matched to the intraoperative respective 
bone or bone fragment through using a surface model of the bone or bone fragment and 
its projections in the planes of the fluoroscopic images; 
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c) fluoroscopy based systems, which use calibrated fluoroscopes to generate 
undistorted images of a bone or bone fragment and virtual geometric representations of 
the projection of surgical tools. 

A method of generating tomographic images of a body using penetrating radiation is 
known from EP 1 004 272 LIN. This known method comprises the steps of 

- cycling the radiation source among a plurality of positions relative to the region of 
interest of the body such that radiation from the radiation source passing through each 
of a plurality of focal planes, which are parallel to the detector plane and within the 
region of interest, impinge upon the detector plane superimposed and offset from each 
other; 

- Shifting the electronic views for a first selected focal plane, such that the radiation 
which passes through each incremental element of the first selected focal plane 
contributes to a common pixel of the electronic views; and 

- Finally, summing the electronic views such that the pixels of each electronic view 
corresponding to the incremental element on the first selected focal plane are summed 
to generate a slice image taken through the first selected focal plane. 

This known method has the disadvantage that CT-scanning delivers cross-sectional 
images of the patient body whereas fluoroscopic X-ray images deliver two-dimensional 
images and therefore many X-ray images are required in several parallel planes and 
under several different angles of the radiation source to obtain a three-dimensional 
model. 

On this point, the invention intends to provide remedial measures. The invention is 
based on the objective of providing a method that allows to establish a virtual three- 
dimensional representation of a bone or bone fragment based on fluoroscopic images. 
Therewith: 

- the radiation exposure is reduced; 

- three-dimensional images instead of projected two-dimensional images may be used; 
and 

- closed (minimally invasive) surgical operation in reality is turned to open surgical 
operation in a virtual world. 
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The invention solves the posed problem with a method that displays the features of 
claim 1. 

The method according to the invention concerns the establishment of a virtual three- 
dimensional representation (3D model) from X-ray images and essentially comprises 

A) positioning an X-ray device such that the region of interest at the patient's body may 
be mapped on the plane of projection of the X-ray device; 

B) taking at least one medical image of the region of interest mapped on the plane of 
projection and storing this first image as a set of data in a data storage means of a 
computer. Today, image data acquisition is computerized and the images are delivered 
in digital form as a matrix of currently typical 128 2 to 1024 2 picture elements; 

C) display the at least one medical image at a display means connected to the 
computer, whereby the display means may be the display of the computer or e.g. a 
head mounted display; 

D) creating a three-dimensional virtual representation of the bone or bone fragment by 
means of using a control means connected to the computer; and 

E) adjusting the size of the three-dimensional virtual representation to the size of the 
bone or bone fragment by means of using the control means. 

The creation and adjustment of the virtual three-dimensional representation using the 
control means may be performed visually through using a keyboard or virtual keyboard, 
a mouse or a pointer as control means and determining the shape and dimensions 
visually at the display means. 

The preferred application of the method according to the invention further comprises the 
steps of: 

F) taking a second medical image of a patient's bone or bone fragment at an angle with 
respect to the first medical image; 

G) storing the second medical image as a set of data in the data storage means of the 
computer; 

H) display the first and second medical image at the display means; 

I) creating a three-dimensional virtual representation of the bone or bone fragment by 
means of using a control means connected to the computer; and 
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K) adjusting the size of the three-dimensional virtual representation to the size of the 
bone or bone fragment by means of using the control means. 

Furthermore, the method according to the invention preferably comprises the step of 
reconstructing an axis, e.g. the central axis of the at least one bone or bone fragment by 

- defining respective points on the at least one medical image represented on the 
display means, e.g. the display of the computer. The definition of these points may be 
performed e.g. with a keyboard or virtual keyboard, a mouse or a pointer. Alternatively, 
an automatic axis identification by a computing means might be considered, e.g. by 
chamfer methods; 

- automatic surface or volume methods (chamfer methods); identification by a 
computing means might be considered, e.g. by chamfer methods; or 

- identification of anatomical points. 

The size of the virtual representation is thereby adjusted until the projection boundary 
size of the virtual representation is as close to the projected image boundary of the 

The reconstruction of an axis of the bone or bone fragment that is to be represented 
through the respective virtual representation is achieved by reconstructing two points on 
the specific axis. Each point is reconstructed through choosing projection points from at 
least two acquired images. Preferably, the computer is provided with an algorithm that 
allows the projection of the X-ray beam for each selected point on the other images 
through a thin line. This allows verification of the selected point and facilitates the 
identification of the other point that must be chosen on the displayed line. The definition 
and adjustment of the virtual representation is preferably performed in a perspective 
representation determined from the projected images taken with the C-arm X-ray 
device. After the virtual representation is defined and adjusted to the images of the bone 
or bone fragments a check is performed whether the projected images of the bone or 
bone fragment are covered by the projected boundary of the virtual representation 
which is calculated through a specific C-arm X-ray projection model. Thereby, the 
projected images correspond to the images taken by means of the C-arm X-ray device 
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under step B) and F). Since the virtual representation is defined from images of the 
bone or bone fragment that have a different axis of projection a three-dimensional 
representation with a central axis coinciding with the longitudinal axis of the bone or 
bone fragment may be established. Therewith, a more precise manipulation of the bone 
or bone fragments is possible. 

In a further development of the method according to the invention it comprises 
representing surgical tools and surgical implants at the display means connected to the 
computer together with the virtual representations of the respective bone or bone 
fragments. Furthermore, in order to plan the surgical action by means of particular 
interface means, e.g. a keyboard or virtual keyboard, a mouse or a pointer the method 
allows displacing the virtual bone representations, the surgical tools and surgical 
implants on the display means. Therefore, geometrical models of the surgical tools and 
implants that are provided by their manufacturers may be used. 

Furthermore, the method according to the invention may be used as a planning method 
for a subsequent surgical action using surgical navigation to perform the surgical action. 

Therefore, in order to use surgical navigation to perform the surgical action as planned 
with the above steps the method further comprises the steps of 

A) attaching a reference means at each bone or bone fragment involved in the surgical 
action; 

B) measuring the position and orientation of each reference means with respect to a 
system of coordinates by means of a position measurement device; 

C) measuring the position and orientation of a third reference means fixedly attached to 
a C-arm X-ray device having an axis of projection and whereof the position and 
orientation of the plane of projection with respect to the system of coordinates defined 
by the third reference means is known; and 

D) measuring the position and orientation of a fourth reference means fixedly attached 
to a surgical tool or implant. 

Fluoroscopy based systems allow superimposed real-time virtual line graphics 
visualization of the projection of surgical tools and bone fragments relative to the 
acquired C-arm X-ray images. Therefore, a comprehensive calibration procedure has to 
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be applied where all physical parameters are determined that specify imaging attributes. 
These parameters are then recalled and combined with the C-arm X-ray device's 
current spatial position and orientation with respect to the patient's anatomy. 
Advantageously, the pre-calibration of the C-arm X-ray device is divided into three 
steps: extrinsic, intrinsic and mechanic calibration. The extrinsic calibration provides 
external geometric parameters describing the X-ray projection such as the initial focal 
point (X-ray emitter) and image plane positions. The intrinsic calibration extracts 
parameters quantifying the distortions caused by the electronic optics of the image 
intensifier. It allows a correction for these distortions in every acquired image. As most 
C-arm X-ray devices are subject to significant elastic frame deformations when their 
position is changed, a mechanic calibration compensates the related variations of the 
focal point. The deformations of the C-arm X-ray device are mapped over the whole 
range of motion to be able to interpolate between these values. 

The reference means preferably comprise at least three markers that are non-collinearly 
arranged. The markers as well as the detectors of the position measurement device 
may be acoustic or electromagnetic effective means such as energy emitting, receiving 
or reflecting means. For instance as energy emitting means: 

- Light sources, particularly light emitting diodes (LED's); 

- Infrared light emitting diodes (IRED's); or 

- Acoustic transmitters 

or as energy receiving means: 

- Photodiodes; or 

- Microphones 

may be used. Other position measurement devices contain coils as energy emitting 
means and Hall-effect components as energy receiving means may be used as well. 

Each reference means defines a local system of coordinates with a fixed mathematical 
relation to every point of the body e.g. the bone, bone fragment, surgical tool or surgical 
implant where the reference means is attached to. The position measurement device 
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may be connected to a computer. By means of the position measurement device 
coordinate transformations between any of the above systems of coordinates may be 
performed. A medical image taken by means of the C-arm X-ray device reflects to 
momentary position of the bone or bone fragment and must therefore be registered to 
each reference means attached at the bone or bone fragments. Therefore, matrices 
allowing coordinate transformations between the system of coordinates of the reference 
means at the C-arm X-ray device and the reference means attached at the bone or 
bone fragments are obtained by measuring the positions and orientations by means of 
the position measurement device and then stored at the acquisition time of the 
respective image in the data storage means of the computer. 

In case of optoelectronic position measurement, a custom optoelectronic position 
measurement device may be used e.g. an OPTOTRAK 3020 System, Northern Digital, 
Waterloo, On., Canada. It preferably comprises 

- an OPTOTRAK 3020 Position Sensor consisting of three one-dimensional charge- 
coupled devices (CCD) paired with three lens cells and mounted on a stabilised bar. 
Within each of the three lens cells, light from an infrared marker is directed onto a CCD 
and measured. All three measurements together determine - in real time - the three- 
dimensional location of the marker; 

- a system control unit; 

- a computer interface card and cables; 

- data collection and display software; and 

- a strober and marker kit. 

The advantages achieved by the invention are essentially to be seen in the fact that, 
thanks to the method according to the invention a three-dimensional representation of a 
bone or bone fragment may be established from two-dimensional C-arm X-ray images. 
This allows a higher precision when manipulating the bone or bone fragment during the 
surgical action than the present use of two-dimensional images produced by the C-arm 
X-ray device. Furthermore, the necessary amount of fluoroscopic images to be taken of 
the patient's body and therewith the time of exposure to radiation may be significantly 
reduced. It allows an optimal selection of the shape of surgical implants that shall be 
implanted into the patient's body by means of the planned surgical action e.g. size and 
length of an intramedullary nail, bending of a plate or position of screws. 
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The method according to the invention is explained in even more detail with reference to 
the partially schematic illustrations. 

Shown are: 

Fig. 1 the performance of the method according to the invention by means of a C-arm 
X-ray device connected to a computer; 

Fig. 2 the establishment of the virtual three-dimensional representation by means of a 
computer; 

Fig. 3 the performance of a computer assisted close reduction of a fractured bone; and 

Fig. 4 the adjustment of the virtual three-dimensional representation to a bone or bone 
fragment using the method according to the invention. 

Fig. 1 depicts the steps of positioning a mobile X-ray device 6 (C-arm X-ray device) in 
different positions with respect to a bone 21 such that a region of interest of the bone 21 
is mappable under different angles of view on the plane of projection 7 of the X-ray 
device 6, taking a first and second image 11;13 (Fig. 2) and storing the images 11;13 as 
a first and second set of data in a data storage means 9 of the computer 10. The 
images 11;13 which are taken may be displayed at a display means 32 which is e.g. the 
display 20 of the computer 10. Other display means such as a head mounted display 
may be used as well. 

Optionally, a position measurement device 4 may be applied. By means of the position 
measurement device 4 the position of the markers 29 (Fig. 3) attached at the reference 
means 5 at the X-ray device 6 with respect to an on-site three-dimensional system of 
coordinates 3 may be measured. This additional step allows to determine the position 
and orientation of the plane of projection 7,7' in each selected position of the X-ray 
device 6 with respect to the on-site system of coordinates 3. If the selected virtual 
representation is a symmetrical representation as e.g. a virtual cylinder coaxial to the 
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specified axis (Fig. 2) of the bone 21 the position measurement of the X-ray device 6 is 
not necessary. 

Fig. 2 depicts the establishment of the three-dimensional representation at two images 
1 1 ;1 3 of a fractured bone 21, whereby the image 11 is taken at an anterior-posterior 
view and the image 13 is taken at a lateral-medial view of the region of interest of the 
bone 21. Using a mouse as interface means 26 a first point 14';14" and a second point 
ISMS" may be selected in each image 11;13 therewith specifying a distinctive axis 
16'; 16" of the bone 21. Once the specific axis 16 of the bone 21 is determined a virtual 
representation is selected. Here, the virtual representation is a virtual cylinder 17, 
whereby the anterior-posterior projected virtual cylinder 17" is represented on the first 
image 11 and the lateral-medial projected virtual cylinder 17' is represented on the 
second image 13. With a mouse as interface means 26 the size of the virtual cylinder 
17 may be adjusted to the size of the bone 21 by adjusting one or both of the projected 
virtual cylinders 17'; 17" to the displayed bone 21. 

Fig. 3 depicts a computer assisted closed reduction of a fractured bone 21. After a 
virtual representation of the proximal and distal bone fragments 18; 19 is established 
and displayed at e.g. the display 20 of the computer 10 a set of image data of a surgical 
tool 22, e.g. a drill drive is loaded into the processor 27 (Fig. 2) of the computer 10. A 
forth reference means 28 is attached at the drill drive so that the relative position of the 
drill drive with respect to the first and second reference means 1;2 may be determined 
by measuring the position of the markers 29 on each of the reference means 1 ;2;28 with 
respect to the three-dimensional system of coordinates 3 by means of the position 
measurement device 4. 

In order to register the images 1 1 ;13 taken by the X-ray device 6 the surgeon manually 
identifies an anatomical landmark at the bone fragment 18; 19 on the two images 1 1 ;1 3 
(Fig. 4) represented at a display means and then the related three-dimensional position 
of the anatomical landmark relative to the corresponding reference base 1; 2 is 
determined. This step is performed through coordinate transformations between the 
system of coordinates affixed to the images, the system of coordinates affixed to the 
plane of projection 7, the system of coordinates affixed to the third reference means 5 
attached at the X-ray device 6 and the on-site system of coordinates 3. 
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Such that the virtual representation may be used to intraoperatively visualize the 
position of the virtual representation directly and based on the measured positions of 
the bone fragments 18; 19. Therewith the surgeon may perform the alignment of the 
bone fragments 18; 19 with direct visual control. As the surgeon displaces the bone 
fragment 18; 19 the corresponding virtual representation is moved as well on the display 
20. Similarly, a surgical tool 22 or surgical implant 23, e.g. an intramedullary nail may be 
intraoperatively guided with direct visual control at the display 20. 
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Claims 

1 . Method for establishing a virtual three-dimensional representation of a bone or bone 
fragment from X-ray images comprising the steps of 

A) taking at least one medical image (1 1) of a patient's bone (21) or bone fragment; 

B) storing the at least one medical image (1 1) as a set of data in a data storage means 
(9) of a computer (10); 

characterized in that the method further comprises the steps of 

C) display the at least one medical image (1 1 ) at a display means (32) connected to the 
computer (10); 

D) creating a three-dimensional virtual representation of the bone (21) or bone fragment 
by means of using a control means (26) connected to the computer (10); and 

E) adjusting the size of the three-dimensional virtual representation to the size of the 
bone (21) or bone fragment by means of using the control means (26). 

2. Method according to claim 1, characterized in that it further comprises the steps of: 

F) taking a second medical image (13) of a patient's bone (21) or bone fragment at an 
angle with respect to the at least one medical image (1 1 ); 

G) storing the second medical image (13) as a set of data in the data storage means (9) 
of the computer (10); 

H) display the first and second medical image (11;13) at a display means (32); 

I) creating a three-dimensional virtual representation of the bone (21) or bone fragment 
by means of using a control means (26) connected to the computer (10); and 

K) adjusting the size of the three-dimensional virtual representation to the size of the 
bone (21) or bone fragment by means of using the control means (26). 

3. Method according to claim 1 or 2, characterized in that it further comprises the step of 
reconstructing an axis (16) of the bone (21) or bone fragment (1 8;1 9) at the display 
means (32) by using a control means (26) and by defining at least two respective points 
(14;15) on at least one of the images (11;13) represented on the display means (32). 
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4. Method according to claim 1 or 2, characterised in that it further comprises the step of 
reconstructing an axis (16) of the bone (21) or bone fragment (18; 19) through an 
automatic axis identification. 

5. Method according to claim 3 or 4, characterized in that the virtual representation 
comprises a enveloping surface (25) of a bone (21) or bone fragment (18; 19), said 
enveloping surface (25) being aligned to the axis (16). 

6. Method according to claim 3 or 4, characterised in that the virtual representation 
comprises a body representing a bone (21) or bone fragment (18; 19), said body being 
aligned to the axis (16). 

7. Method according to claim 1 , characterised in that it further comprises the steps of 

a) loading a set of image data of a surgical tool (22) into a processor (27) of the 
computer (10); 

b) representing the surgical tool (22) at the display means (32) together with the virtual 
representation of a respective bone (21) or bone fragment (18;19); and 

c) displacing the virtual representation and/or the representation of the surgical tool (22) 
on the display means (32) by using a control means (26) connected to the computer 
(10).. 

8. Method according to claim 1 or 7, characterised in that it further comprises the steps 
of 

d) loading a set of image data of a surgical implant (23) into the processor (27) of the 
computer (10); 

e) representing the surgical implant (23) at the display means (32) together with the 
virtual representation of a respective bone (21) or bone fragments (18;19); and 

f) displacing the virtual representation and the representation of the surgical implant (23) 
on the display means (32) by using a control means (26). 

9. Method according to one of the claims 1 to 8, characterised in that it further 
comprises the step of rotating the virtual representation on the display means (32) by 
using a control means (26) in order to receive a perspective view of the region of 
interest of a bone (21) or bone fragments (18; 19). 
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10. Method according to claim 8 or 9, characterised in that it further comprises the step 
of simulating a surgical action at the display means (32) by means of adjusting the 
virtual representation and arranging the surgical implant (23) at its desired position. 

11. Method according to one of the claims 8 to 10, characterised in that it further 
comprises the steps of 

g) attaching a reference means (1;2) at each bone (21) or bone fragment (18;19) of 
interest; 

h) measuring the position and orientation of each reference means (1 ;2) with respect to 
a system of coordinates (3) by means of a position measurement device (4); 

i) measuring the position and orientation of a further reference means (5) fixedly 
attached to a mobile X-ray device (6) having an axis of projection (12) and a plane of 
projection (7), whereby the position and orientation of said plane of projection (7) with 
respect to the system of coordinates (8) defined by the third reference means (5) is 
known; and 

j) measuring the position and orientation of a fourth reference means (28) fixedly 
attached to a surgical tool (22). 
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